Skip to content

Import GitHub OSV advisories incrementally - #2498

Open
Prachi194agrawal wants to merge 1 commit into
aboutcode-org:mainfrom
Prachi194agrawal:fix/2317-incremental-github-osv-imports
Open

Prachi194agrawal wants to merge 1 commit into
aboutcode-org:mainfrom
Prachi194agrawal:fix/2317-incremental-github-osv-imports

Conversation

@Prachi194agrawal

Copy link
Copy Markdown

Related to #2317

The GitHub OSV importer currently scans and processes every reviewed advisory on each run. This PR adds an incremental import prototype that processes added and changed advisory files after the first successful import.

The importer stores the last successfully imported commit in a GitImportCheckpoint for the pipeline and repository. On later runs, it compares that commit with the current checkout and selects changed JSON files. If the previous commit is unavailable or the history has diverged, it falls back to a full import.

Deleted advisory paths are logged for review, while existing records and their history are retained. The checkpoint advances only after importing the selected advisories and cleaning up the checkout successfully. Failed runs keep the previous checkpoint so the changes can be retried.

This prototype reduces advisory parsing and database work for github_osv_importer_v2. It retains the existing clone process. Deletion handling and checkpoint timing can be reviewed before extending the approach to other importers.

Testing

The tests use a local Git repository to cover first and incremental imports, unchanged and deleted files, missing or rewritten history, renames, and failure and retry behavior.

venv/bin/python -m pytest \
  vulnerabilities/tests/pipelines/v2_importers/test_github_osv_importer_v2.py \
  --reuse-db -q

Result: 15 passed, 5 warnings.

The new migration was also applied locally with venv/bin/python manage.py migrate.
Screenshot from 2026-11-12 22-21-54

Signed-off-by: Prachi Agrawal <prachi194agrawal@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant